BLD: migrate build backend to meson-python and enable abi3 wheels by neutrinoceros · Pull Request #556 · pydata/bottleneck

neutrinoceros · 2026-05-30T16:09:13Z

~~This is based off #511~~
close #516
very much a work in progress.

When attempting a build with

uvx meson setup build
uvx meson compile -C build

I get 2 errors about malloc and free being undefined. Here's the first one:

bottleneck/src/reduce_template.c:819:5: error: call to undeclared library function 'malloc' with type 'void *(unsigned long)'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  819 |     BUFFER_NEW(float64, LENGTH)
      |     ^
bottleneck/src/reduce_template.c:802:31: note: expanded from macro 'BUFFER_NEW'
  802 |         npy_##dtype *buffer = malloc(length * sizeof(npy_##dtype));
      |                               ^
bottleneck/src/reduce_template.c:819:5: note: include the header <stdlib.h> or explicitly provide a declaration for 'malloc'

I don't understand how this happens, #include <stdlib.h> is already present in bottleneck/src/reduce_template.c.

If I run uv build instead, I get

../meson.build:25:3: ERROR: File bottleneck/src/reduce.c does not exist.

so clearly I must be doing something wrong, but I'm not too surprised because this is the first time I attempt to hook python scripts into a meson build.

neutrinoceros · 2026-05-31T08:30:01Z

I reached a somewhat functional state where at least some jobs are passing.
What's broken:

docs: many references to setup.py (critical)
mnt: Makefile and bn_setup.sh should be either removed or updated (tbd)
warnings-as-errors in tests (very important)
dynamic config from setuptools (possibly not needed, but I need to understand what I'm giving up)
dynamic version number (nice to have, but I'm not personally motivated to preserve this)

neutrinoceros · 2026-05-31T10:22:22Z

looks like this is what's failing windows-arm64 jobs: astral-sh/uv#12906

neutrinoceros · 2026-05-31T10:40:04Z

got a bit further using the workaround described in the issue I just linked. Now meson correctly detects a native Python interpreter, but then hit

C:\a\_temp\uv-python-dir\cpython-3.11.15-windows-aarch64-none\python311.dll:
      file not recognized: file format not recognized

which I don't understand yet

rgommers

I had a quick looks; this looks like a good start to me.

rgommers · 2026-05-31T16:40:51Z

+  'nonreduce_axis',
+  [
+    bn_config_h,
+    custom_target(


If you define a generator, this can be a one-liner per file, same as in for example:

https://github.com/scipy/scipy/blob/20c4b5e75727ae1d3dc19a2bb8c65c43f1366592/scipy/sparse/meson.build#L7-L8

a generator is how I initially tried to write this but I struggled to define a template for the output filename, because it's obtained by subtraction rather than addition.

Could rename the source files? They're not valid C code, so .c.in or .c.template seems more accurate anyway

Yes, I was thinking doing just that as a cleanup step after I got CI stable again. Though I prefer extensions like .template.c, as it allows syntax highlighting to sort-of work instead of not at all.

actually went for it tonight because I had limited bandwidth and this didn't require much. I ended up using your suggestion (.c.template) because I couldn't work out how to implement .template.c using available substitutions .

neutrinoceros · 2026-06-01T08:33:41Z

I'm hesitant to just drop windows-arm64 as a normal CI target. It works in the context of cibw but I'm struggling to set it up using actions/setup-python and/or astral-sh/setup-uv.

neutrinoceros · 2026-06-02T07:58:05Z

I got lost as I tried to get everything working at the same time. I've disabled limited-api builds for now because I found it hard to setup correctly with 3.14t also being a target. I'm also disabling failing targets as I go right now, with the goal to reveal the subset that does work already, before I tackle failures one by one again.

rgommers · 2026-09-12T15:50:44Z

Oh wait, the wheel build issue does ring a bell: the problem here is that we're building all Windows wheels in a single job (which none of the other projects I work on do; it leads to issues like this one), and CPython didn't fix a free-threading header issue - that's why the 3.11 to 3.14 builds work and 3.14t chokes. From https://py-free-threading.github.io/installing-cpython/:

"Due to limitations of the Windows Python.org installer, using free-threaded Python installed from the Python.org installer may lead to trouble. In particular, if you install both a free-threaded and gil-enabled build of Python 3.13+ using the Python.org installer, both installs will share a site-packages folder. This can very quickly lead to broken environments if packages for both versions are simultaneously installed."

Let me figure out if there's an easy workaround.

rgommers · 2026-09-12T16:13:08Z

@neutrinoceros the problem is config-settings = { build-dir = "build" I think: that reuses the exact same build directory across all builds on win-arm64. That won't work in this case, because numpy is now installed in a different location. I think it's a recipe for problems. Separate jobs would fix it, or separate build directories. Reusing build dirs between jobs never seems healthy to me.

rgommers · 2026-09-12T19:38:59Z

Those fixups don't look right. I thought about it some more, and reusing a build directory is just not a valid thing to do between two fresh builds with different dependencies.

build-dir = "build" should be deleted from pyproject.toml. That way meson-python will simply create a fresh build dir, avoiding the problem. I think restructuring the jobs to have a single build per job is a bit more work, but better - it's a lot easier to understand, and safer.

I'll note that setuptools happens to work in this case, because within the build dir it does namespacing like so:

build/
    cp313-win_arm64/
    cp314-win_arm64/
    cp314t-win_arm64/

that's special-casing the interpreter, which makes it work - but in general it's also susceptible to changes between builds that aren't special-cased (e.g., a change to a compiler, codegen, etc.).

Meson, as a general-purpose build system, treats Python as any other dependency and not one with special-casing for how build caching and meson setup --reconfigure work.

neutrinoceros · 2026-09-13T09:07:58Z

I must not be understanding where the meson executable is even found. Is it included in the docker image ?
No, meson is a dependency of meson-python.

Let me clarify: the issue I'm seeing is that despite having meson==1.11.2 locked, uv is still somehow installing 1.12.0 in CI. I initially assume uv would respect my constraint, which meant meson 1.12.0 had to come from somewhere else, but having turned on uv's debug log I can see that the problem is really there.

neutrinoceros · 2026-09-13T09:11:06Z

Ah ! I think that's just because the build isolation intentionally ignores runtime constraints. Promoting the constraint to an actual build-time requirement should do the trick.

neutrinoceros · 2026-09-13T10:18:12Z

Okay wheels are working now.
The remaining errors are all on regular windows CI, and relate to interpreter selection in at least 3 different ways:

cp315 + x64combos are either unsupported by actions/setup-python, or misconfigured

The version '3.15' with architecture 'x64' was not found for Windows 2022.

build-time failures for x86

Need python for x86_64, but found x86
        Run-time dependency python found: NO  (tried pkgconfig and sysconfig)
  
        ..\meson.build:43:3: ERROR: Python dependency not found

A linker error on windows arm64

2026-09-13T10:08:26.4415431Z       C:\hostedtoolcache\windows\Python\3.11.9\arm64\python311.dll: file not
2026-09-13T10:08:26.4415776Z       recognized: file format not recognized
2026-09-13T10:08:26.4416050Z       collect2.exe: error: ld returned 1 exit status

I think 1 and 2 are relatively simple to fix. I'll give it a try now.

neutrinoceros · 2026-09-13T10:33:30Z

I got x64 jobs to work by downgrading Python to 3.14
I don't really understand the difference between x86 and x86_64, any chance this is a shortcoming from meson itself ?
I still have no idea how to fix the linker error on arm64

rgommers · 2026-09-13T11:23:30Z

2. I don't really understand the difference between x86 and x86_64, any chance this is a shortcoming from meson itself ?

x86 is a 32-bit interpreter. Need python for x86_64, but found x86 is the standard issue where we need/use the bus1/cabuild action to activate 32-bit MSVC, except in reverse - so the job is using that action when it shouldn't.

I still have no idea how to fix the linker error on arm64

It's picking up MinGW:

        C compiler for the host machine: cc (gcc 14.2.0 "cc
        (x86_64-posix-seh-rev2, Built by MinGW-Builds project) 14.2.0")

Pass --vsenv to the build to force MSVC, like in the wheel build jobs.

cp315 + x64combos are either unsupported by actions/setup-python, or misconfigured

Needs a pre-release set to true probably?

neutrinoceros · 2026-09-13T12:37:41Z

Thanks !

arm64 is go
windows CI is back to Python 3.15

x86 is a 32-bit interpreter. Need python for x86_64, but found x86 is the standard issue where we need/use the bus1/cabuild action to activate 32-bit MSVC, except in reverse - so the job is using that action when it shouldn't.

But bus1/cabuild isn't used at all in regular CI, it's just there for cibw, so I'm still confused. On the off chance that you got it mixed up, I tried adding the action to regular CI, and the error message is different, but it's still failing.

rgommers · 2026-09-13T13:26:27Z

      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
        with:
          persist-credentials: false
+      - name: Set up Python ${{ matrix.python-version }}


Note that this duplicates a Python install from ./.github/bottleneck-action with setup-uv.

rgommers · 2026-09-13T13:28:49Z

Try inserting this in test-windows (copied from numpy):

      - name: pkg-config
        run: |
          python -m pip install pkgconf==2.5.1.post1

It's going wrong in the Python detection on x86-64, and the CI log has this warning:

 WARNING: Found pkg-config 'C:\\Strawberry\\perl\\bin\\pkg-config.BAT'

The windows-latest image has a completely broken pkg-config on the PATH and for some reason that I don't know about, GitHub doesn't get that fixed.

neutrinoceros · 2026-09-13T15:35:10Z

Interesting. The errors we get now look furiously like the ones we saw with bus1/cabuild

neutrinoceros · 2026-09-14T09:51:05Z

CI is almost there, though I don't think I'll have time to work on this in the next 2 weeks. If anyone wants to pick it up in the mean time be my guest.

rgommers · 2026-09-14T10:07:43Z

Thanks for the push. I'll try to find some time to figure out why this wasn't easier. It really should be; there might be some weird cibuildwheel/meson/abi3 interaction that made this more complicated than it is for numpy and scipy.

turn off limited api builds for cp315t

neutrinoceros force-pushed the bld/meson branch 3 times, most recently from 679039b to fd49952 Compare May 31, 2026 08:25

This was referenced May 31, 2026

WHL: expand build matrix #557

Merged

WHL: fix wheel uploads #558

Merged

neutrinoceros force-pushed the bld/meson branch from d1101f0 to 4d2a4b4 Compare May 31, 2026 10:11

neutrinoceros mentioned this pull request May 31, 2026

WHL: switch build frontend from build[uv] to uv #561

Merged

neutrinoceros force-pushed the bld/meson branch 4 times, most recently from ad2a594 to 64d6587 Compare May 31, 2026 16:28

rgommers reviewed May 31, 2026

View reviewed changes

neutrinoceros force-pushed the bld/meson branch 5 times, most recently from 11ff03d to fc3f675 Compare June 1, 2026 08:16

neutrinoceros force-pushed the bld/meson branch 8 times, most recently from 7f54ef5 to f995b9d Compare June 2, 2026 07:35

neutrinoceros force-pushed the bld/meson branch 2 times, most recently from 0c4ff54 to bdd5f3f Compare September 13, 2026 09:01

neutrinoceros force-pushed the bld/meson branch from bd052df to 7e81855 Compare September 13, 2026 09:09

neutrinoceros force-pushed the bld/meson branch 3 times, most recently from 76deb7d to 9e51d2d Compare September 13, 2026 10:06

neutrinoceros force-pushed the bld/meson branch from d2839f2 to b30aa37 Compare September 13, 2026 12:39

rgommers reviewed Sep 13, 2026

View reviewed changes

neutrinoceros force-pushed the bld/meson branch 3 times, most recently from 97c1c0c to 7197ab3 Compare September 13, 2026 15:52

neutrinoceros added 5 commits September 14, 2026 12:41

BLD: migrate build backend to meson, enable abi3 wheels

98ac4dd

turn off limited api builds for cp315t

temp: cleanup dyn ver

28fc41f

draft fix for windows CI

e55ba04

tentative fix build (quick and dirty)

db2f9c1

exp: constraint on meson

eccbc19

neutrinoceros force-pushed the bld/meson branch from f09dbf1 to eccbc19 Compare September 14, 2026 10:41

Uh oh!

Conversation

neutrinoceros commented May 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neutrinoceros commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neutrinoceros commented May 31, 2026

Uh oh!

neutrinoceros commented May 31, 2026

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rgommers May 31, 2026

Choose a reason for hiding this comment

Uh oh!

neutrinoceros May 31, 2026

Choose a reason for hiding this comment

Uh oh!

rgommers May 31, 2026

Choose a reason for hiding this comment

Uh oh!

neutrinoceros May 31, 2026

Choose a reason for hiding this comment

Uh oh!

neutrinoceros May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

neutrinoceros commented Jun 1, 2026

Uh oh!

neutrinoceros commented Jun 2, 2026

Uh oh!

rgommers commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Sep 12, 2026

Uh oh!

rgommers commented Sep 12, 2026

Uh oh!

neutrinoceros commented Sep 13, 2026

Uh oh!

neutrinoceros commented Sep 13, 2026

Uh oh!

neutrinoceros commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neutrinoceros commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Sep 13, 2026

Uh oh!

neutrinoceros commented Sep 13, 2026

Uh oh!

rgommers Sep 13, 2026

Choose a reason for hiding this comment

Uh oh!

rgommers commented Sep 13, 2026

Uh oh!

neutrinoceros commented Sep 13, 2026

Uh oh!

neutrinoceros commented Sep 14, 2026

Uh oh!

rgommers commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

neutrinoceros commented May 30, 2026 •

edited

Loading

neutrinoceros commented May 31, 2026 •

edited

Loading

neutrinoceros May 31, 2026 •

edited

Loading

rgommers commented Sep 12, 2026 •

edited

Loading

neutrinoceros commented Sep 13, 2026 •

edited

Loading

neutrinoceros commented Sep 13, 2026 •

edited

Loading